{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c94724aa",
   "metadata": {
    "id": "c94724aa"
   },
   "source": [
    "## Arxiv Search with OpenCLIP\n",
    "\n",
    "In this example, we’ll create an Arxiv search or recommender system using multimodal semantic search powered by LanceDB. We’ll also compare its results with keyword-based search on Nomic's Atlas."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "XGYJJdKmGvU_",
   "metadata": {
    "id": "XGYJJdKmGvU_"
   },
   "source": [
    "## OpenCLIP\n",
    "\n",
    "OpenCLIP is an open-source version of CLIP, a model that links images and text. It learns to understand both by training on pairs of pictures and descriptions. You can use it to:\n",
    "\n",
    "- Match images with text (like finding pictures based on a description).\n",
    "- Classify images without extra training (zero-shot learning).\n",
    "- Build creative tools (e.g., text-to-image models).\n",
    "\n",
    "It’s flexible, free to use, and works well for tasks combining images and language.\n",
    "\n",
    "![CLIP (1)](https://github.com/lancedb/vectordb-recipes/assets/15766192/11b3b900-0bcb-4a4a-8fd4-804611c85972)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0gLg0dghIHl3",
   "metadata": {
    "id": "0gLg0dghIHl3"
   },
   "source": [
    "Now let's jump into building Arxiv papers recommendation system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d29cbe5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7d29cbe5",
    "outputId": "20ee70d4-cde1-4440-e09f-bda2920acff3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m115.1/115.1 kB\u001b[0m \u001b[31m1.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.5/1.5 MB\u001b[0m \u001b[31m12.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.6/21.6 MB\u001b[0m \u001b[31m45.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.4/53.4 kB\u001b[0m \u001b[31m7.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.2/2.2 MB\u001b[0m \u001b[31m107.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m81.1/81.1 kB\u001b[0m \u001b[31m13.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m98.7/98.7 kB\u001b[0m \u001b[31m9.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Building wheel for sgmllib3k (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
     ]
    }
   ],
   "source": [
    "# install required dependencies\n",
    "!pip install lancedb open_clip_torch arxiv pandas --q"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88cba25e",
   "metadata": {
    "id": "88cba25e"
   },
   "source": [
    "### Embedding Paper Summary with OpenCLIP\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fba615af",
   "metadata": {
    "id": "fba615af"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import open_clip\n",
    "import pandas as pd\n",
    "from open_clip import tokenizer\n",
    "from tqdm import tqdm\n",
    "from collections import defaultdict\n",
    "import arxiv\n",
    "import lancedb\n",
    "\n",
    "\n",
    "def embed_func_clip(text):\n",
    "    \"\"\"\n",
    "    Helper function which returns embeddings of text and images using OpenCLIP.\n",
    "    \"\"\"\n",
    "    model, _, preprocess = open_clip.create_model_and_transforms(\n",
    "        \"ViT-B-32\", pretrained=\"laion2b_s34b_b79k\"\n",
    "    )\n",
    "    tokenizer = open_clip.get_tokenizer(\"ViT-B-32\")\n",
    "    with torch.no_grad():\n",
    "        text_features = model.encode_text(tokenizer(text))\n",
    "    return text_features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5802574",
   "metadata": {
    "id": "e5802574"
   },
   "source": [
    "### Create a DataFrame of the desired length\n",
    "\n",
    "Here we'll use arxiv python utility to interact with **Arxiv API** and get the document data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb8afe10",
   "metadata": {
    "id": "eb8afe10"
   },
   "outputs": [],
   "source": [
    "def get_arxiv_df(embed_func, length=100):\n",
    "    results = arxiv.Search(\n",
    "        query=\"cat:cs.AI OR cat:cs.CV OR cat:stat.ML\",\n",
    "        max_results=length,\n",
    "        sort_by=arxiv.SortCriterion.Relevance,\n",
    "        sort_order=arxiv.SortOrder.Descending,\n",
    "    ).results()\n",
    "    df = defaultdict(list)\n",
    "    for result in tqdm(results, total=length):\n",
    "        try:\n",
    "            df[\"title\"].append(result.title)\n",
    "            df[\"summary\"].append(result.summary)\n",
    "            df[\"authors\"].append(str(result.authors))\n",
    "            df[\"url\"].append(result.entry_id)\n",
    "            df[\"vector\"].append(embed_func(result.summary).tolist()[0])\n",
    "\n",
    "        except Exception as e:\n",
    "            print(\"error: \", e)\n",
    "\n",
    "    return pd.DataFrame(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa2edccf",
   "metadata": {
    "id": "aa2edccf"
   },
   "outputs": [],
   "source": [
    "LENGTH = 100  # Reduce the size for demo\n",
    "\n",
    "\n",
    "# create lancedb table\n",
    "def create_table():\n",
    "    db = lancedb.connect(\"db\")\n",
    "    df = get_arxiv_df(embed_func_clip, LENGTH)\n",
    "\n",
    "    tbl = db.create_table(\"arxiv\", data=df, mode=\"overwrite\")\n",
    "\n",
    "    return tbl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "760efa67-8742-4087-91c8-49465b4843b0",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 213,
     "referenced_widgets": [
      "cc0868346dd946a6bd15c95859bbeed0",
      "72c68a5aa5b047a281c7801c365844c0",
      "bc68fb9820154bbaae3bb09fd05bd0bc",
      "2f28629727b54c48955215485757d7d8",
      "8b9b5405fd60414d9d6cc4d4af611b0c",
      "6373e2b3100346d9adf4825b518efcba",
      "25c0946ae6d04f33b60eceaefb7d87e4",
      "d1de9b1b3ccd49a89a316cda336c8a91",
      "e9927513616a4bd391ff97c6ff3b2f29",
      "821875e1743c4609b1678418af9c526e",
      "e60575abd83742e6ad1ab90071eaf7cf"
     ]
    },
    "id": "760efa67-8742-4087-91c8-49465b4843b0",
    "outputId": "ea067d70-1168-4d2f-daf9-e9a7e08af58b"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-4-32cd16380017>:7: DeprecationWarning: The 'Search.results' method is deprecated, use 'Client.results' instead\n",
      "  ).results()\n",
      "  0%|          | 0/100 [00:00<?, ?it/s]/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:88: UserWarning: \n",
      "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
      "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
      "You will be able to reuse this secret in all of your notebooks.\n",
      "Please note that authentication is recommended but still optional to access public models or datasets.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cc0868346dd946a6bd15c95859bbeed0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "open_clip_pytorch_model.bin:   0%|          | 0.00/605M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 100/100 [02:57<00:00,  1.77s/it]\n"
     ]
    }
   ],
   "source": [
    "tbl = create_table()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9c04cb6",
   "metadata": {
    "id": "e9c04cb6"
   },
   "outputs": [],
   "source": [
    "import lancedb\n",
    "\n",
    "db = lancedb.connect(\"db\")\n",
    "\n",
    "if \"arxiv\" not in db.table_names():\n",
    "    tbl = create_table()\n",
    "else:\n",
    "    tbl = db.open_table(\"arxiv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d51fadf3-3f98-44fa-9ecd-0dbdfb2f9eb7",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "d51fadf3-3f98-44fa-9ecd-0dbdfb2f9eb7",
    "outputId": "bff0649e-2da6-4362-e8e2-f92713ef9132"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tbl)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09adb9d3",
   "metadata": {
    "id": "09adb9d3"
   },
   "source": [
    "## Semantic Searching by **Concepts** or **Summary**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acc38daa",
   "metadata": {
    "id": "acc38daa"
   },
   "outputs": [],
   "source": [
    "from IPython.display import display, HTML\n",
    "\n",
    "\n",
    "def search_table(query, embed_func=embed_func_clip, lim=3):\n",
    "    db = lancedb.connect(\"db\")\n",
    "    tbl = db.open_table(\"arxiv\")\n",
    "\n",
    "    embs = embed_func(query)\n",
    "\n",
    "    return tbl.search(embs.tolist()[0]).limit(3).to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27391ad7-433e-4b32-8215-534d47de08d8",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "27391ad7-433e-4b32-8215-534d47de08d8",
    "outputId": "afe51537-e90f-4b4b-a086-9db324d49125"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tbl)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "BnWfIXaAIT8q",
   "metadata": {
    "id": "BnWfIXaAIT8q"
   },
   "source": [
    "### Start Querying"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "971be6ef",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 629
    },
    "id": "971be6ef",
    "outputId": "e54e95f9-abda-4477-e9f0-99ceafd2f925"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>summary</th>\n",
       "      <th>authors</th>\n",
       "      <th>url</th>\n",
       "      <th>_distance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>XFlow: Cross-modal Deep Neural Networks for Audiovisual Classification</td>\n",
       "      <td>In recent years, there have been numerous developments towards solving\\nmultimodal tasks, aiming to learn a stronger representation than through a\\nsingle modality. Certain aspects of the data can be particularly useful in this\\ncase - for example, correlations in the space or time domain across modalities\\n- but should be wisely exploited in order to benefit from their full predictive\\npotential. We propose two deep learning architectures with multimodal\\ncross-connections that allow for dataflow between several feature extractors\\n(XFlow). Our models derive more interpretable features and achieve better\\nperformances than models which do not exchange representations, usefully\\nexploiting correlations between audio and visual data, which have a different\\ndimensionality and are nontrivially exchangeable. Our work improves on existing\\nmultimodal deep learning algorithms in two essential ways: (1) it presents a\\nnovel method for performing cross-modality (before features are learned from\\nindividual modalities) and (2) extends the previously proposed\\ncross-connections which only transfer information between streams that process\\ncompatible data. Illustrating some of the representations learned by the\\nconnections, we analyse their contribution to the increase in discrimination\\nability and reveal their compatibility with a lip-reading network intermediate\\nrepresentation. We provide the research community with Digits, a new dataset\\nconsisting of three data types extracted from videos of people saying the\\ndigits 0-9. Results show that both cross-modal architectures outperform their\\nbaselines (by up to 11.5%) when evaluated on the AVletters, CUAVE and Digits\\ndatasets, achieving state-of-the-art results.</td>\n",
       "      <td>[arxiv.Result.Author('Cătălina Cangea'), arxiv.Result.Author('Petar Veličković'), arxiv.Result.Author('Pietro Liò')]</td>\n",
       "      <td>http://arxiv.org/abs/1709.00572v2</td>\n",
       "      <td>40.346901</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dualing GANs</td>\n",
       "      <td>Generative adversarial nets (GANs) are a promising technique for modeling a\\ndistribution from samples. It is however well known that GAN training suffers\\nfrom instability due to the nature of its maximin formulation. In this paper,\\nwe explore ways to tackle the instability problem by dualizing the\\ndiscriminator. We start from linear discriminators in which case conjugate\\nduality provides a mechanism to reformulate the saddle point objective into a\\nmaximization problem, such that both the generator and the discriminator of\\nthis 'dualing GAN' act in concert. We then demonstrate how to extend this\\nintuition to non-linear formulations. For GANs with linear discriminators our\\napproach is able to remove the instability in training, while for GANs with\\nnonlinear discriminators our approach provides an alternative to the commonly\\nused GAN training algorithm.</td>\n",
       "      <td>[arxiv.Result.Author('Yujia Li'), arxiv.Result.Author('Alexander Schwing'), arxiv.Result.Author('Kuan-Chieh Wang'), arxiv.Result.Author('Richard Zemel')]</td>\n",
       "      <td>http://arxiv.org/abs/1706.06216v1</td>\n",
       "      <td>40.449284</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Domain Generalization for Object Recognition with Multi-task Autoencoders</td>\n",
       "      <td>The problem of domain generalization is to take knowledge acquired from a\\nnumber of related domains where training data is available, and to then\\nsuccessfully apply it to previously unseen domains. We propose a new feature\\nlearning algorithm, Multi-Task Autoencoder (MTAE), that provides good\\ngeneralization performance for cross-domain object recognition.\\n  Our algorithm extends the standard denoising autoencoder framework by\\nsubstituting artificially induced corruption with naturally occurring\\ninter-domain variability in the appearance of objects. Instead of\\nreconstructing images from noisy versions, MTAE learns to transform the\\noriginal image into analogs in multiple related domains. It thereby learns\\nfeatures that are robust to variations across domains. The learnt features are\\nthen used as inputs to a classifier.\\n  We evaluated the performance of the algorithm on benchmark image recognition\\ndatasets, where the task is to learn features from multiple datasets and to\\nthen predict the image label from unseen datasets. We found that (denoising)\\nMTAE outperforms alternative autoencoder-based models as well as the current\\nstate-of-the-art algorithms for domain generalization.</td>\n",
       "      <td>[arxiv.Result.Author('Muhammad Ghifary'), arxiv.Result.Author('W. Bastiaan Kleijn'), arxiv.Result.Author('Mengjie Zhang'), arxiv.Result.Author('David Balduzzi')]</td>\n",
       "      <td>http://arxiv.org/abs/1508.07680v1</td>\n",
       "      <td>41.127644</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Example 1: MobileSAM paper abstract 2nd half\n",
    "\n",
    "query = \"\"\"\n",
    "Many of such applications need to be run on resource-constraint edge devices,\n",
    "like mobile phones. In this work, we aim to make SAM mobile-friendly by replacing the heavyweight\n",
    "image encoder with a lightweight one. A naive way to train such a new SAM as in the original SAM\n",
    "paper leads to unsatisfactory performance, especially when limited training sources are available. We\n",
    "find that this is mainly caused by the coupled optimization of the image encoder and mask decoder,\n",
    "motivated by which we propose decoupled distillation. Concretely, we distill the knowledge from\n",
    "the heavy image encoder (ViT-H in the original SAM) to a lightweight image encoder, which can be\n",
    "automatically compatible with the mask decoder in the original SAM. The training can be completed\n",
    "on a single GPU within less than one day, and the resulting lightweight SAM is termed MobileSAM\n",
    "which is more than 60 times smaller yet performs on par with the original SAM. For inference speed,\n",
    "With a single GPU, MobileSAM runs around 10ms per image: 8ms on the image encoder and 4ms\n",
    "on the mask decoder. With superior performance, our MobileSAM is around 5 times faster than the\n",
    "concurrent FastSAM and 7 times smaller, making it more suitable for mobile applications. Moreover,\n",
    "we show that MobileSAM can run relatively smoothly on CPU\n",
    "\"\"\"\n",
    "\n",
    "result = search_table(query)\n",
    "\n",
    "result.pop(\"vector\")\n",
    "display(HTML(result.to_html()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4ccd273",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 577
    },
    "id": "f4ccd273",
    "outputId": "e0afdb9b-2578-418a-e37f-1fd19fb44251"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>summary</th>\n",
       "      <th>authors</th>\n",
       "      <th>url</th>\n",
       "      <th>_distance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A General Theory for Training Learning Machine</td>\n",
       "      <td>Though the deep learning is pushing the machine learning to a new stage,\\nbasic theories of machine learning are still limited. The principle of\\nlearning, the role of the a prior knowledge, the role of neuron bias, and the\\nbasis for choosing neural transfer function and cost function, etc., are still\\nfar from clear. In this paper, we present a general theoretical framework for\\nmachine learning. We classify the prior knowledge into common and\\nproblem-dependent parts, and consider that the aim of learning is to maximally\\nincorporate them. The principle we suggested for maximizing the former is the\\ndesign risk minimization principle, while the neural transfer function, the\\ncost function, as well as pretreatment of samples, are endowed with the role\\nfor maximizing the latter. The role of the neuron bias is explained from a\\ndifferent angle. We develop a Monte Carlo algorithm to establish the\\ninput-output responses, and we control the input-output sensitivity of a\\nlearning machine by controlling that of individual neurons. Applications of\\nfunction approaching and smoothing, pattern recognition and classification, are\\nprovided to illustrate how to train general learning machines based on our\\ntheory and algorithm. Our method may in addition induce new applications, such\\nas the transductive inference.</td>\n",
       "      <td>[arxiv.Result.Author('Hong Zhao')]</td>\n",
       "      <td>http://arxiv.org/abs/1704.06885v1</td>\n",
       "      <td>33.708359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Learning Visual Reasoning Without Strong Priors</td>\n",
       "      <td>Achieving artificial visual reasoning - the ability to answer image-related\\nquestions which require a multi-step, high-level process - is an important step\\ntowards artificial general intelligence. This multi-modal task requires\\nlearning a question-dependent, structured reasoning process over images from\\nlanguage. Standard deep learning approaches tend to exploit biases in the data\\nrather than learn this underlying structure, while leading methods learn to\\nvisually reason successfully but are hand-crafted for reasoning. We show that a\\ngeneral-purpose, Conditional Batch Normalization approach achieves\\nstate-of-the-art results on the CLEVR Visual Reasoning benchmark with a 2.4%\\nerror rate. We outperform the next best end-to-end method (4.5%) and even\\nmethods that use extra supervision (3.1%). We probe our model to shed light on\\nhow it reasons, showing it has learned a question-dependent, multi-step\\nprocess. Previous work has operated under the assumption that visual reasoning\\ncalls for a specialized architecture, but we show that a general architecture\\nwith proper conditioning can learn to visually reason effectively.</td>\n",
       "      <td>[arxiv.Result.Author('Ethan Perez'), arxiv.Result.Author('Harm de Vries'), arxiv.Result.Author('Florian Strub'), arxiv.Result.Author('Vincent Dumoulin'), arxiv.Result.Author('Aaron Courville')]</td>\n",
       "      <td>http://arxiv.org/abs/1707.03017v5</td>\n",
       "      <td>36.282284</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Encoder Based Lifelong Learning</td>\n",
       "      <td>This paper introduces a new lifelong learning solution where a single model\\nis trained for a sequence of tasks. The main challenge that vision systems face\\nin this context is catastrophic forgetting: as they tend to adapt to the most\\nrecently seen task, they lose performance on the tasks that were learned\\npreviously. Our method aims at preserving the knowledge of the previous tasks\\nwhile learning a new one by using autoencoders. For each task, an\\nunder-complete autoencoder is learned, capturing the features that are crucial\\nfor its achievement. When a new task is presented to the system, we prevent the\\nreconstructions of the features with these autoencoders from changing, which\\nhas the effect of preserving the information on which the previous tasks are\\nmainly relying. At the same time, the features are given space to adjust to the\\nmost recent environment as only their projection into a low dimension\\nsubmanifold is controlled. The proposed system is evaluated on image\\nclassification tasks and shows a reduction of forgetting over the\\nstate-of-the-art</td>\n",
       "      <td>[arxiv.Result.Author('Amal Rannen Triki'), arxiv.Result.Author('Rahaf Aljundi'), arxiv.Result.Author('Mathew B. Blaschko'), arxiv.Result.Author('Tinne Tuytelaars')]</td>\n",
       "      <td>http://arxiv.org/abs/1704.01920v1</td>\n",
       "      <td>37.254250</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Exmaple 2: Search via a concept you're reading\n",
    "query = \"\"\"\n",
    "What is the general idea behind self-supervised learning.\n",
    "\"\"\"\n",
    "\n",
    "result = search_table(query)\n",
    "\n",
    "result.pop(\"vector\")\n",
    "display(HTML(result.to_html()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f471b55",
   "metadata": {
    "id": "5f471b55"
   },
   "source": [
    "### Full Text Search\n",
    "In text retrieval, full-text search refers to techniques for searching a single computer-stored document or a collection in a full-text database. Full-text search is distinguished from searches based on metadata or on parts of the original texts represented in databases\n",
    "\n",
    "LanceDB now provides **experimental** support for full text search. This is currently Python only. We plan to push the integration down to Rust in the future to make this available for JS as well.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "FKPSsbpq5Weq",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FKPSsbpq5Weq",
    "outputId": "3d61b95d-5ca3-47c5-f9b3-a7dd9f91b9ab"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting tantivy\n",
      "  Downloading tantivy-0.21.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.2 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.2/4.2 MB\u001b[0m \u001b[31m17.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: tantivy\n",
      "Successfully installed tantivy-0.21.0\n"
     ]
    }
   ],
   "source": [
    "!pip install tantivy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2d693cf",
   "metadata": {
    "id": "b2d693cf"
   },
   "source": [
    "### Build FTS index for the summary\n",
    "Here, we're building the FTS index using python bindings for tantivy. You can also build the index for any other text column. A full-text index stores information about significant words and their location within one or more columns of a database table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c383b22",
   "metadata": {
    "id": "5c383b22"
   },
   "outputs": [],
   "source": [
    "# This cell might take a few mins\n",
    "tbl.create_fts_index(\"summary\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4116b06",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "f4116b06",
    "outputId": "37a78abd-514f-48b3-9aab-9f4c01f91d38"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>summary</th>\n",
       "      <th>authors</th>\n",
       "      <th>url</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Expert Gate: Lifelong Learning with a Network of Experts</td>\n",
       "      <td>In this paper we introduce a model of lifelong learning, based on a Network\\nof Experts. New tasks / experts are learned and added to the model\\nsequentially, building on what was learned before. To ensure scalability of\\nthis process,data from previous tasks cannot be stored and hence is not\\navailable when learning a new task. A critical issue in such context, not\\naddressed in the literature so far, relates to the decision which expert to\\ndeploy at test time. We introduce a set of gating autoencoders that learn a\\nrepresentation for the task at hand, and, at test time, automatically forward\\nthe test sample to the relevant expert. This also brings memory efficiency as\\nonly one expert network has to be loaded into memory at any given time.\\nFurther, the autoencoders inherently capture the relatedness of one task to\\nanother, based on which the most relevant prior model to be used for training a\\nnew expert, with finetuning or learning without-forgetting, can be selected. We\\nevaluate our method on image classification and video prediction problems.</td>\n",
       "      <td>[arxiv.Result.Author('Rahaf Aljundi'), arxiv.Result.Author('Punarjay Chakravarty'), arxiv.Result.Author('Tinne Tuytelaars')]</td>\n",
       "      <td>http://arxiv.org/abs/1611.06194v2</td>\n",
       "      <td>4.703215</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Approximate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs</td>\n",
       "      <td>The idea of computer vision as the Bayesian inverse problem to computer\\ngraphics has a long history and an appealing elegance, but it has proved\\ndifficult to directly implement. Instead, most vision tasks are approached via\\ncomplex bottom-up processing pipelines. Here we show that it is possible to\\nwrite short, simple probabilistic graphics programs that define flexible\\ngenerative models and to automatically invert them to interpret real-world\\nimages. Generative probabilistic graphics programs consist of a stochastic\\nscene generator, a renderer based on graphics software, a stochastic likelihood\\nmodel linking the renderer's output and the data, and latent variables that\\nadjust the fidelity of the renderer and the tolerance of the likelihood model.\\nRepresentations and algorithms from computer graphics, originally designed to\\nproduce high-quality images, are instead used as the deterministic backbone for\\nhighly approximate and stochastic generative models. This formulation combines\\nprobabilistic programming, computer graphics, and approximate Bayesian\\ncomputation, and depends only on general-purpose, automatic inference\\ntechniques. We describe two applications: reading sequences of degraded and\\nadversarially obscured alphanumeric characters, and inferring 3D road models\\nfrom vehicle-mounted camera images. Each of the probabilistic graphics programs\\nwe present relies on under 20 lines of probabilistic code, and supports\\naccurate, approximately Bayesian inferences about ambiguous real-world images.</td>\n",
       "      <td>[arxiv.Result.Author('Vikash K. Mansinghka'), arxiv.Result.Author('Tejas D. Kulkarni'), arxiv.Result.Author('Yura N. Perov'), arxiv.Result.Author('Joshua B. Tenenbaum')]</td>\n",
       "      <td>http://arxiv.org/abs/1307.0060v1</td>\n",
       "      <td>4.515473</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Learning Visual Reasoning Without Strong Priors</td>\n",
       "      <td>Achieving artificial visual reasoning - the ability to answer image-related\\nquestions which require a multi-step, high-level process - is an important step\\ntowards artificial general intelligence. This multi-modal task requires\\nlearning a question-dependent, structured reasoning process over images from\\nlanguage. Standard deep learning approaches tend to exploit biases in the data\\nrather than learn this underlying structure, while leading methods learn to\\nvisually reason successfully but are hand-crafted for reasoning. We show that a\\ngeneral-purpose, Conditional Batch Normalization approach achieves\\nstate-of-the-art results on the CLEVR Visual Reasoning benchmark with a 2.4%\\nerror rate. We outperform the next best end-to-end method (4.5%) and even\\nmethods that use extra supervision (3.1%). We probe our model to shed light on\\nhow it reasons, showing it has learned a question-dependent, multi-step\\nprocess. Previous work has operated under the assumption that visual reasoning\\ncalls for a specialized architecture, but we show that a general architecture\\nwith proper conditioning can learn to visually reason effectively.</td>\n",
       "      <td>[arxiv.Result.Author('Ethan Perez'), arxiv.Result.Author('Harm de Vries'), arxiv.Result.Author('Florian Strub'), arxiv.Result.Author('Vincent Dumoulin'), arxiv.Result.Author('Aaron Courville')]</td>\n",
       "      <td>http://arxiv.org/abs/1707.03017v5</td>\n",
       "      <td>4.332870</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Memory Aware Synapses: Learning what (not) to forget</td>\n",
       "      <td>Humans can learn in a continuous manner. Old rarely utilized knowledge can be\\noverwritten by new incoming information while important, frequently used\\nknowledge is prevented from being erased. In artificial learning systems,\\nlifelong learning so far has focused mainly on accumulating knowledge over\\ntasks and overcoming catastrophic forgetting. In this paper, we argue that,\\ngiven the limited model capacity and the unlimited new information to be\\nlearned, knowledge has to be preserved or erased selectively. Inspired by\\nneuroplasticity, we propose a novel approach for lifelong learning, coined\\nMemory Aware Synapses (MAS). It computes the importance of the parameters of a\\nneural network in an unsupervised and online manner. Given a new sample which\\nis fed to the network, MAS accumulates an importance measure for each parameter\\nof the network, based on how sensitive the predicted output function is to a\\nchange in this parameter. When learning a new task, changes to important\\nparameters can then be penalized, effectively preventing important knowledge\\nrelated to previous tasks from being overwritten. Further, we show an\\ninteresting connection between a local version of our method and Hebb's\\nrule,which is a model for the learning process in the brain. We test our method\\non a sequence of object recognition tasks and on the challenging problem of\\nlearning an embedding for predicting $&lt;$subject, predicate, object$&gt;$ triplets.\\nWe show state-of-the-art performance and, for the first time, the ability to\\nadapt the importance of the parameters based on unlabeled data towards what the\\nnetwork needs (not) to forget, which may vary depending on test conditions.</td>\n",
       "      <td>[arxiv.Result.Author('Rahaf Aljundi'), arxiv.Result.Author('Francesca Babiloni'), arxiv.Result.Author('Mohamed Elhoseiny'), arxiv.Result.Author('Marcus Rohrbach'), arxiv.Result.Author('Tinne Tuytelaars')]</td>\n",
       "      <td>http://arxiv.org/abs/1711.09601v4</td>\n",
       "      <td>4.307245</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Explaining Aviation Safety Incidents Using Deep Temporal Multiple Instance Learning</td>\n",
       "      <td>Although aviation accidents are rare, safety incidents occur more frequently\\nand require a careful analysis to detect and mitigate risks in a timely manner.\\nAnalyzing safety incidents using operational data and producing event-based\\nexplanations is invaluable to airline companies as well as to governing\\norganizations such as the Federal Aviation Administration (FAA) in the United\\nStates. However, this task is challenging because of the complexity involved in\\nmining multi-dimensional heterogeneous time series data, the lack of\\ntime-step-wise annotation of events in a flight, and the lack of scalable tools\\nto perform analysis over a large number of events. In this work, we propose a\\nprecursor mining algorithm that identifies events in the multidimensional time\\nseries that are correlated with the safety incident. Precursors are valuable to\\nsystems health and safety monitoring and in explaining and forecasting safety\\nincidents. Current methods suffer from poor scalability to high dimensional\\ntime series data and are inefficient in capturing temporal behavior. We propose\\nan approach by combining multiple-instance learning (MIL) and deep recurrent\\nneural networks (DRNN) to take advantage of MIL's ability to learn using weakly\\nsupervised data and DRNN's ability to model temporal behavior. We describe the\\nalgorithm, the data, the intuition behind taking a MIL approach, and a\\ncomparative analysis of the proposed algorithm with baseline models. We also\\ndiscuss the application to a real-world aviation safety problem using data from\\na commercial airline company and discuss the model's abilities and\\nshortcomings, with some final remarks about possible deployment directions.</td>\n",
       "      <td>[arxiv.Result.Author('Vijay Manikandan Janakiraman')]</td>\n",
       "      <td>http://arxiv.org/abs/1710.04749v2</td>\n",
       "      <td>4.206257</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>A General Theory for Training Learning Machine</td>\n",
       "      <td>Though the deep learning is pushing the machine learning to a new stage,\\nbasic theories of machine learning are still limited. The principle of\\nlearning, the role of the a prior knowledge, the role of neuron bias, and the\\nbasis for choosing neural transfer function and cost function, etc., are still\\nfar from clear. In this paper, we present a general theoretical framework for\\nmachine learning. We classify the prior knowledge into common and\\nproblem-dependent parts, and consider that the aim of learning is to maximally\\nincorporate them. The principle we suggested for maximizing the former is the\\ndesign risk minimization principle, while the neural transfer function, the\\ncost function, as well as pretreatment of samples, are endowed with the role\\nfor maximizing the latter. The role of the neuron bias is explained from a\\ndifferent angle. We develop a Monte Carlo algorithm to establish the\\ninput-output responses, and we control the input-output sensitivity of a\\nlearning machine by controlling that of individual neurons. Applications of\\nfunction approaching and smoothing, pattern recognition and classification, are\\nprovided to illustrate how to train general learning machines based on our\\ntheory and algorithm. Our method may in addition induce new applications, such\\nas the transductive inference.</td>\n",
       "      <td>[arxiv.Result.Author('Hong Zhao')]</td>\n",
       "      <td>http://arxiv.org/abs/1704.06885v1</td>\n",
       "      <td>4.150894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>A Brief Survey of Deep Reinforcement Learning</td>\n",
       "      <td>Deep reinforcement learning is poised to revolutionise the field of AI and\\nrepresents a step towards building autonomous systems with a higher level\\nunderstanding of the visual world. Currently, deep learning is enabling\\nreinforcement learning to scale to problems that were previously intractable,\\nsuch as learning to play video games directly from pixels. Deep reinforcement\\nlearning algorithms are also applied to robotics, allowing control policies for\\nrobots to be learned directly from camera inputs in the real world. In this\\nsurvey, we begin with an introduction to the general field of reinforcement\\nlearning, then progress to the main streams of value-based and policy-based\\nmethods. Our survey will cover central algorithms in deep reinforcement\\nlearning, including the deep $Q$-network, trust region policy optimisation, and\\nasynchronous advantage actor-critic. In parallel, we highlight the unique\\nadvantages of deep neural networks, focusing on visual understanding via\\nreinforcement learning. To conclude, we describe several current areas of\\nresearch within the field.</td>\n",
       "      <td>[arxiv.Result.Author('Kai Arulkumaran'), arxiv.Result.Author('Marc Peter Deisenroth'), arxiv.Result.Author('Miles Brundage'), arxiv.Result.Author('Anil Anthony Bharath')]</td>\n",
       "      <td>http://arxiv.org/abs/1708.05866v2</td>\n",
       "      <td>3.549962</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Interpretable Explanations of Black Boxes by Meaningful Perturbation</td>\n",
       "      <td>As machine learning algorithms are increasingly applied to high impact yet\\nhigh risk tasks, such as medical diagnosis or autonomous driving, it is\\ncritical that researchers can explain how such algorithms arrived at their\\npredictions. In recent years, a number of image saliency methods have been\\ndeveloped to summarize where highly complex neural networks \"look\" in an image\\nfor evidence for their predictions. However, these techniques are limited by\\ntheir heuristic nature and architectural constraints. In this paper, we make\\ntwo main contributions: First, we propose a general framework for learning\\ndifferent kinds of explanations for any black box algorithm. Second, we\\nspecialise the framework to find the part of an image most responsible for a\\nclassifier decision. Unlike previous works, our method is model-agnostic and\\ntestable because it is grounded in explicit and interpretable image\\nperturbations.</td>\n",
       "      <td>[arxiv.Result.Author('Ruth Fong'), arxiv.Result.Author('Andrea Vedaldi')]</td>\n",
       "      <td>http://arxiv.org/abs/1704.03296v4</td>\n",
       "      <td>3.451381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Self corrective Perturbations for Semantic Segmentation and Classification</td>\n",
       "      <td>Convolutional Neural Networks have been a subject of great importance over\\nthe past decade and great strides have been made in their utility for producing\\nstate of the art performance in many computer vision problems. However, the\\nbehavior of deep networks is yet to be fully understood and is still an active\\narea of research. In this work, we present an intriguing behavior: pre-trained\\nCNNs can be made to improve their predictions by structurally perturbing the\\ninput. We observe that these perturbations - referred as Guided Perturbations -\\nenable a trained network to improve its prediction performance without any\\nlearning or change in network weights. We perform various ablative experiments\\nto understand how these perturbations affect the local context and feature\\nrepresentations. Furthermore, we demonstrate that this idea can improve\\nperformance of several existing approaches on semantic segmentation and scene\\nlabeling tasks on the PASCAL VOC dataset and supervised classification tasks on\\nMNIST and CIFAR10 datasets.</td>\n",
       "      <td>[arxiv.Result.Author('Swami Sankaranarayanan'), arxiv.Result.Author('Arpit Jain'), arxiv.Result.Author('Ser Nam Lim')]</td>\n",
       "      <td>http://arxiv.org/abs/1703.07928v2</td>\n",
       "      <td>3.417501</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Graph Approximation and Clustering on a Budget</td>\n",
       "      <td>We consider the problem of learning from a similarity matrix (such as\\nspectral clustering and lowd imensional embedding), when computing pairwise\\nsimilarities are costly, and only a limited number of entries can be observed.\\nWe provide a theoretical analysis using standard notions of graph\\napproximation, significantly generalizing previous results (which focused on\\nspectral clustering with two clusters). We also propose a new algorithmic\\napproach based on adaptive sampling, which experimentally matches or improves\\non previous methods, while being considerably more general and computationally\\ncheaper.</td>\n",
       "      <td>[arxiv.Result.Author('Ethan Fetaya'), arxiv.Result.Author('Ohad Shamir'), arxiv.Result.Author('Shimon Ullman')]</td>\n",
       "      <td>http://arxiv.org/abs/1406.2602v1</td>\n",
       "      <td>3.358721</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## FTS via title\n",
    "result = (\n",
    "    tbl.search(\"What is the general idea behind self-supervised learning.\")\n",
    "    .limit(10)\n",
    "    .to_pandas()\n",
    ")\n",
    "\n",
    "result.pop(\"vector\")\n",
    "\n",
    "display(HTML(result.to_html()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06850511",
   "metadata": {
    "id": "06850511"
   },
   "source": [
    "## Analysing OpenCLIP embeddings on Nomic\n",
    "\n",
    "Atlas is a platform for interacting with both small and internet scale unstructured datasets.\n",
    "\n",
    "Atlas enables you to:\n",
    "* Store, update and organize multi-million point datasets of unstructured text, images and embeddings.\n",
    "* Visually interact with embeddings of your data from a web browser.\n",
    "* Operate over unstructured data and embeddings with topic modeling, semantic duplicate clustering and semantic search.\n",
    "* Generate high dimensional and two-dimensional embeddings of your data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e94d1855",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "e94d1855",
    "outputId": "acf94cfb-a582-4df3-8e8b-889707dcd068"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/41.2 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m41.2/41.2 kB\u001b[0m \u001b[31m1.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.5/62.5 kB\u001b[0m \u001b[31m3.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Building wheel for nomic (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
     ]
    }
   ],
   "source": [
    "!pip install nomic --q"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8U9G-B-Z5yPF",
   "metadata": {
    "id": "8U9G-B-Z5yPF"
   },
   "source": [
    "### Nomic Login\n",
    "\n",
    "We are using Nomic to use Atlas for visualizing dataset in clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f5b3933",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3f5b3933",
    "outputId": "bfeda0f4-9a33-45f0-8dc3-6fbf90de7f9b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m                                  \u001b[0m\u001b[1mAuthenticate with the Nomic API\u001b[0m\u001b[1m                                   \u001b[0m\n",
      "\u001b[1m                                  \u001b[0m\u001b[4;94mhttps://atlas.nomic.ai/cli-login\u001b[0m\u001b[1m                                  \u001b[0m\n",
      "\u001b[1m       \u001b[0m\u001b[1mClick the above link to retrieve your access token and then run `nomic login \u001b[0m\u001b[1m[\u001b[0m\u001b[1mtoken\u001b[0m\u001b[1m]\u001b[0m\u001b[1m`\u001b[0m\u001b[1m        \u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!nomic login"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50e63ef8",
   "metadata": {
    "id": "50e63ef8"
   },
   "outputs": [],
   "source": [
    "!nomic login [token] # Paste your token from Nomic Ai cli login -- here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea123fc4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ea123fc4",
    "outputId": "05a672a9-0c00-4f92-9bb5-c4254eb7ad3b"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m2024-02-25 06:18:17.433\u001b[0m | \u001b[33m\u001b[1mWARNING \u001b[0m | \u001b[36mnomic.atlas\u001b[0m:\u001b[36mmap_data\u001b[0m:\u001b[36m96\u001b[0m - \u001b[33m\u001b[1mAn ID field was not specified in your data so one was generated for you in insertion order.\u001b[0m\n",
      "\u001b[32m2024-02-25 06:18:19.434\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mnomic.dataset\u001b[0m:\u001b[36m_create_project\u001b[0m:\u001b[36m868\u001b[0m - \u001b[1mCreating dataset `inquisitive-jaynes`\u001b[0m\n",
      "\u001b[32m2024-02-25 06:18:19.719\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mnomic.atlas\u001b[0m:\u001b[36mmap_data\u001b[0m:\u001b[36m108\u001b[0m - \u001b[1mUploading data to Atlas.\u001b[0m\n",
      "1it [00:00,  1.62it/s]\n",
      "\u001b[32m2024-02-25 06:18:20.370\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mnomic.dataset\u001b[0m:\u001b[36m_add_data\u001b[0m:\u001b[36m1536\u001b[0m - \u001b[1mUpload succeeded.\u001b[0m\n",
      "\u001b[32m2024-02-25 06:18:20.374\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mnomic.atlas\u001b[0m:\u001b[36mmap_data\u001b[0m:\u001b[36m123\u001b[0m - \u001b[1m`prasantdixit9876/inquisitive-jaynes`: Data upload succeeded to dataset`\u001b[0m\n",
      "\u001b[32m2024-02-25 06:18:20.655\u001b[0m | \u001b[33m\u001b[1mWARNING \u001b[0m | \u001b[36mnomic.dataset\u001b[0m:\u001b[36mcreate_index\u001b[0m:\u001b[36m1121\u001b[0m - \u001b[33m\u001b[1mYou did not specify the `topic_label_field` option in your topic_model, your dataset will not contain auto-labeled topics.\u001b[0m\n",
      "\u001b[32m2024-02-25 06:18:21.396\u001b[0m | \u001b[1mINFO    \u001b[0m | \u001b[36mnomic.dataset\u001b[0m:\u001b[36mcreate_index\u001b[0m:\u001b[36m1245\u001b[0m - \u001b[1mCreated map `inquisitive-jaynes` in dataset `prasantdixit9876/inquisitive-jaynes`: https://atlas.nomic.ai/data/prasantdixit9876/inquisitive-jaynes/map\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from nomic import atlas\n",
    "import numpy as np\n",
    "\n",
    "# Get pandas dataframe from lancedb table\n",
    "df = tbl.to_pandas()\n",
    "\n",
    "# get embeddings from df\n",
    "embs = np.array(df.pop(\"vector\").to_list())\n",
    "\n",
    "project = atlas.map_data(embeddings=embs, data=df.to_dict(\"records\"))\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a17553eb",
   "metadata": {
    "id": "a17553eb"
   },
   "source": [
    "The visualizations are very interesting and is worth exploring more. In preliminary analysis, you can see that it succesfully creates clusters of similar types of papers. There are a few things that can be done next like comparing embeddings on various openclip models sizes and datasets.\n",
    "\n",
    "\n",
    "<img width=\"1433\" alt=\"Screenshot 2023-08-24 at 3 47 51 PM\" src=\"https://github.com/lancedb/vectordb-recipes/assets/15766192/34ef88a3-2925-4450-abcd-1abc350ef3e4\">"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  },
  "vscode": {
   "interpreter": {
    "hash": "511a7c77cb034b09af5465c01316a0f4bb20176d139e60e6d7915f9a637a5037"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
